Skip to content

Descriptions, TTS, Style, and HDR#661

Open
TheJoeFin wants to merge 82 commits into
mainfrom
dev
Open

Descriptions, TTS, Style, and HDR#661
TheJoeFin wants to merge 82 commits into
mainfrom
dev

Conversation

@TheJoeFin

Copy link
Copy Markdown
Owner

PR Classification

New feature and major UI/UX enhancement: unified pattern/recognizer search, voice output, HDR capture correction, and improved settings.

PR Summary

This PR introduces unified "Smart Pattern" (recognizer) and regex search across the app, adds voice output (TTS) features, supports HDR-aware screen capture, and modernizes the settings UI. It also improves Grab Frame and Quick Simple Lookup workflows and adds .tggf file support.

  • Added PatternItem, PatternExecutor, RecognizerExecutor, and integrated Microsoft Recognizers-Text for culture-aware entity recognition (regex/recognizer unification) across Find & Replace, Grab Templates, and Edit Text Window.
  • Replaced legacy search boxes with a shared SearchBar control in Grab Frame and Quick Simple Lookup, supporting pattern/recognizer selection, regex, and exact-match toggles.
  • GrabFrame.xaml/.cs: Added "Speak" toolbar button, speaking progress overlay, customizable toolbar, runtime border style/color switching, .tggf file open/save, and improved freeze/unfreeze logic.
  • FullscreenGrab.SelectionStyles.cs: Enhanced Windows AI Description workflow with spinner overlay, cancel/re-grab/send-to-frame options, and TTS processing announcements.
  • Refactored and modernized all settings pages using SettingsCard/SettingsExpander controls, and added a new Voice Output settings page.

kmcnaught and others added 30 commits May 22, 2026 11:34
Adds a "Speak text" PostGrabAction that reads OCR output aloud using
Windows.Media.SpeechSynthesis. Speaks the final transformed text after
all other actions run in FullscreenGrab; in GrabFrame, speaks only when
the captured text changes to avoid repeating on every OCR tick.

- ITtsEngine interface for future engine swappability
- WindowsSpeechEngine wraps WinRT SpeechSynthesizer + MediaPlayer
- TtsService queues utterances via Channel<string> so new text waits
  rather than interrupting in-progress speech
- TtsSpeakWordLimit setting (default 100) truncates long captures;
  configurable in General Settings
- PostGrabActionManager: new SpeakText_Click action at order 6.6
- GrabFrame: speaks on text change when action is checked
- Tests: count updated to 6, "Speak text" assertion, fire-and-forget test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds SpeakInsteadOfToast setting: when enabled, Text Grab speaks the
captured text aloud rather than chiming a notification toast. Defaults
to enabled.

Includes:
- Stop-speaking button in GrabFrame to cancel playback mid-sentence
- TTS drain-on-shutdown fix so the queue empties cleanly on exit
- Stop() method on TtsService to flush the queue and cancel speech
Adds a dedicated Voice Output page in Settings containing:
- Voice picker populated from SpeechSynthesizer.AllVoices
- Word limit setting
- Speak-instead-of-notification toggle (moved from General Settings)
- Preview button to hear the selected voice

WindowsSpeechEngine now applies the saved TtsVoiceName before
synthesising. TtsVoiceName setting added (empty = system default).
Introduce a new user-scoped boolean setting, WindowsAiDescriptionEnabled, to control Windows AI description features. The setting defaults to False and is added to App.config, Settings.settings, and Settings.Designer.cs.
Added WindowsAiDescriptionLang implementing ILanguage for the new "Windows AI Description" language kind. Introduced GeneratedOcrLinesWords, GeneratedOcrLine, and GeneratedOcrWord classes to generate OCR line/word models from text and bounding boxes. Updated HistoryInfo to support the new language kind.
Introduce WindowsAiDescriptionLang for image captioning via Windows AI. Update LanguageKind, LanguageService, and CaptureLanguageUtilities to support selection and caching. Route OCR/image description requests through new async methods in OcrUtilities and WindowsAiUtilities. Add device capability checks, content moderation, and async polling for image description. Exclude WindowsAiDescriptionLang from table output. Improve resource cleanup and error handling.
Improve test coverage for WindowsAiDescriptionLang integration:
- Save/restore WindowsAiDescriptionEnabled for isolation
- Test inclusion/exclusion in CaptureLanguageUtilities
- Assert correct tag/kind handling in LanguageService
- Verify fallback and HistoryInfo behavior with new language
Introduce WindowsAiDescriptionLang support in LanguagePicker and filter it from the language list. Add a toggle in LanguageSettings to enable/disable Windows AI Description as a language option, with explanatory text. Refactor settings initialization to use a unified loadingLanguageSettings flag and update event handlers accordingly. Implement methods to load and persist the new setting, and ensure language caches are invalidated when toggled.
Add the five Microsoft.Recognizers.Text.* packages (with BUILT-WITH
attribution) and a fixed BuiltInRecognizer catalog -- numbers, ordinals,
percentages, ages, currencies, dimensions, temperatures, dates/times, phone
numbers, emails, URLs, IPs, GUIDs, and booleans -- plus RecognizerExecutor for
matching and applying them (resolved value or matched text, first/last/all/nth).

GrabTemplate gains {r:Name:mode[:value|text][:separator]} placeholders
(TemplateRecognizerMatch), resolved by GrabTemplateExecutor alongside the
existing {p:} regex patterns.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce PatternItem -- a single user-facing "pattern" that wraps either a saved
StoredRegex or a built-in recognizer -- and PatternExecutor, which dispatches
HasMatch/GetMatches/Apply to the regex or recognizer engine. This lets every UI
surface treat the two kinds uniformly. PatternItem also builds the grouped
("Saved Patterns" / "Smart Patterns") choice lists used by the combo pickers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose saved regexes and built-in recognizers together everywhere, split into
"Saved Patterns" and "Smart Patterns" subsections:

- Template inline {...} picker: one Patterns group (InlinePickerItem.Kind drives
  whether selection emits {p:} or {r:}); {p:}/{r:} syntax unchanged.
- Edit Text Window: "Apply Pattern" / "Apply Pattern Per Line" list both kinds and
  transform in place via PatternExecutor.
- Grab Frame: "Search by Pattern" submenu (saved regexes searchable for the first
  time); Quick Simple Lookup and Find & Replace use a unified Patterns combo
  (saved regex loads into the find box so replace still works; recognizers are
  find-only).

Combo pickers use PatternChoice data items with DisplayMemberPath and
non-selectable subsection headers, fixing the selection-box width/height clipping
that came from using ComboBoxItem objects as data. User-visible "Recognizer"
wording is now "Pattern".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A reusable search input used across the search surfaces: a text box with a
placeholder and clear button, a removable smart-pattern chip, a WPF-UI-style
accent focus underline on the whole control, a regex toggle rendered as a
split-button whose chevron opens a scrollable pattern flyout (saved regexes +
smart recognizers), and an optional exact-match toggle.

Adds the ToggleSplitSegment and SplitSegmentButton styles the split-button uses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… Frame

Replaces each window's hand-built search box, regex control, and pattern picker
with the shared SearchBar and routes searching through its SearchChanged event:

- Quick Simple Lookup: swaps its text box, regex toggle, and pattern combo for
  the control; regex validity now flows through SetRegexValidity.
- Find & Replace: the find box becomes a SearchBar; the "Match Exactly" switch
  moves into the box as a toggle (wired via ExactMatchChanged), and a picked
  recognizer can be narrowed by typed text. Adds SetFindText for external callers.
- Grab Frame: the regex, exact-match, and "search by pattern" options that were
  hidden in the text box context menu become visible controls, and the status
  bar grows to fit the taller control.

Retargets RegexManager and EditTextWindow to SetFindText and drops the now-unused
PatternItem combo helpers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Registers a Ctrl+Shift+F key gesture in the Edit Text Window that opens Find &
Replace, and shows the shortcut on the menu item. Ctrl+F stays mapped to
Fullscreen Grab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expanded GetDefaultPatterns summary to clarify exclusion of patterns already handled by built-in recognizers (emails, phones, URLs, IPs, GUIDs, dates, times, currency, numbers). Removed these patterns from the defaults, retaining only credit card, hex color, SSN, and US zip code regexes.
Recognizers: unify saved regexes and smart recognizers as one "Patterns" concept
Introduce a new user-scoped boolean setting, WindowsAiDescriptionEnabled, to control Windows AI description features. The setting defaults to False and is added to App.config, Settings.settings, and Settings.Designer.cs.
Added WindowsAiDescriptionLang implementing ILanguage for the new "Windows AI Description" language kind. Introduced GeneratedOcrLinesWords, GeneratedOcrLine, and GeneratedOcrWord classes to generate OCR line/word models from text and bounding boxes. Updated HistoryInfo to support the new language kind.
Introduce WindowsAiDescriptionLang for image captioning via Windows AI. Update LanguageKind, LanguageService, and CaptureLanguageUtilities to support selection and caching. Route OCR/image description requests through new async methods in OcrUtilities and WindowsAiUtilities. Add device capability checks, content moderation, and async polling for image description. Exclude WindowsAiDescriptionLang from table output. Improve resource cleanup and error handling.
Improve test coverage for WindowsAiDescriptionLang integration:
- Save/restore WindowsAiDescriptionEnabled for isolation
- Test inclusion/exclusion in CaptureLanguageUtilities
- Assert correct tag/kind handling in LanguageService
- Verify fallback and HistoryInfo behavior with new language
Introduce WindowsAiDescriptionLang support in LanguagePicker and filter it from the language list. Add a toggle in LanguageSettings to enable/disable Windows AI Description as a language option, with explanatory text. Refactor settings initialization to use a unified loadingLanguageSettings flag and update event handlers accordingly. Implement methods to load and persist the new setting, and ensure language caches are invalidated when toggled.
Add overloads to GetTextDescriptionWithWinAI for System.Drawing.Bitmap input and CancellationToken support. Use AsTask to enable cancellation of DescribeAsync, remove manual polling, and improve exception handling to propagate OperationCanceledException and log other errors. Temp files are now cleaned up after use.
Introduce a new user flow for Windows AI image description in fullscreen grab mode. The overlay now freezes the selected region as a background, shows a spinner, and after 2 seconds offers Cancel/Re-grab options. If the AI description fails or is empty, a "Send to Grab Frame" option is provided. Refactor PreviousGrabWindow and FullscreenGrab to support this asynchronous, user-driven workflow and improve code organization.
Replace the flat black strip with a centered rounded panel using the
app background, teal border, and drop shadow like the fullscreen grab
toolbar. Buttons now use the TealColor style with icon + label content,
and the bar slides up from the bottom edge with a fade on first reveal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Use C# collection expressions for English word array in WindowsAiUtilities
- Return empty string (not error) if WinAI unavailable for image description
- Ignore single clicks for AI Description in fullscreen grab, reset selection instead
- Cancel grab if overlay closes without user choice to avoid orphaned tasks
- Default TextContent to empty string if OCR result is null
Adds ShowSuccessAsync to PreviousGrabWindow for awaiting overlay close, ensuring text insertion occurs after focus is released. Refactors insertion logic in FullscreenGrab to prevent focus issues, threads cancellation tokens in WindowsAiUtilities, and cleans up redundant code.
Adds a ToggleButton to PreviousGrabWindow for sending OCR results to an Edit Text Window, with visibility and checked state managed during OCR. Introduces a SendToEditTextToggled event and syncs the toggle state across all fullscreen grab windows. Updates running/failed state logic to show or hide the toggle as appropriate.
TheJoeFin and others added 30 commits July 5, 2026 20:25
Integrate HDR screen region capture with tone mapping fallback to GDI when needed. Add Vortice.Direct3D11/DXGI dependencies for HDR support. Refactor ImageMethods to centralize capture logic. Add HdrToneMapper unit tests to validate tone mapping behavior.
Added an "HDR Displays" section to GeneralSettings with a toggle for HDR capture correction and a button to check/grant borderless capture permissions. Updated code-behind to handle toggle state, update settings, and display permission status to the user. Added explanatory text for new options.
Added Bash(git log *) to settings.local.json to permit execution of git log commands via Bash, expanding the set of allowed command patterns.
Added Bash(gh pr view *) and Bash(gh pr diff *) to the allowlist in settings.local.json, enabling use of GitHub CLI commands for viewing and diffing pull requests.
Refactored HDR screen capture to use a process-wide shared D3D11 device, context, and WinRT device for efficiency and reliability. Reduced frame timeout to 250ms to avoid UI stalls. Added helpers for device lifecycle management and improved error handling to recover from device loss. Ensured proper resource cleanup and event unsubscription.
Introduced SettingsCard and SettingsExpander custom controls for consistent settings UI, modeled after Windows Community Toolkit patterns. Refactored ShortcutControl.xaml to use a grid layout for improved alignment and visual consistency; ErrorText is now wrapped, collapsed by default, and spans both columns.
Added SettingsStyles.xaml with styles for settings section headers and page titles. Integrated the new resource dictionary into App.xaml for consistent settings UI appearance.
Refactored all settings pages to use new SettingsCard and SettingsExpander controls, replacing legacy layouts with a more modern, consistent, and accessible design. Grouped related controls with icons, headers, and descriptions for clarity. Updated section headers and page titles for uniformity. No changes to core logic—UI/UX improvements only.
Capture region bitmap before showing overlays (fixes #662). Pass pre-captured bitmap to OCR routines to avoid UI artifacts in OCR input and history. Update method signatures and call sites to support optional pre-captured bitmap.
Added "Bash(git fetch *)" to the allow list in settings.local.json, enabling the use of git fetch in Bash scripts. No other changes were made.
Add tracking of active spreadsheet cell editor and selected text in EditTextWindow. Implement GetSpreadsheetSelectedText to retrieve selection from the cell editor, last known selection, or cell value. Update Find/Replace to use this logic in spreadsheet mode. Handle PreparingCellForEdit to attach selection tracking and ensure cleanup on edit end and selection changes.

closes #663
Introduced ColumnSplitUtilities with support for delimiter, regex, and fixed-length cell splitting, including splitter handling options and robust error handling. Added thorough unit tests covering all split modes and edge cases.
Introduces SplitColumnWindow with UI and logic to split spreadsheet cell contents by delimiter, regex pattern, or fixed length. Features include pattern picker, regex extraction from selection, splitter handling options, live preview, and integration with EditTextWindow for applying splits to selected cells.
Added "Split into Columns…" to the spreadsheet context menu. Implemented handlers to open a split dialog and apply user-defined splitting of selected cell contents into multiple columns, resizing the table as needed.
Updated NCalcAsync, PdfPig, and Microsoft.NET.Test.Sdk to latest versions. Added NuGet.CommandLine (v7.6.0) as a development dependency in all projects. No code changes, only package reference updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Microsoft.Recognizers.Text erroneously declares a dependency on the
NuGet.CommandLine build tool, which pulls nuget.exe into the graph. Keep
a single ExcludeAssets="all" reference in the app project so it satisfies
resolution without flowing any assets into the build output, and drop the
stray references from the test and packaging projects.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The save blanked WordBorderInfoJson, WordBorderInfoFileName, and ImagePath
directly on the supplied instance to keep the serialized metadata clean.
That silently corrupts any caller that passes a shared/live HistoryInfo
rather than a throwaway from AsHistoryItem(). Serialize a shallow copy
instead so the caller's object is left untouched, and add a test asserting
the passed-in fields (including the ImageContent reference) are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… type

The DateTime "values" branch cast the resolution entry to the library's
concrete IEnumerable<Dictionary<string,string>>. A future Recognizers-Text
version storing it as Dictionary<string,object> would fail the cast and
silently fall back to the matched text. Read the collection structurally
(any non-string IEnumerable of dictionaries, string- or object-valued)
instead.

Add FormatResolvedValue tests over synthetic ModelResults pinning both the
current and object-valued shapes, plus real datetime recognizer tests that
lock the live library shape, so a package bump surfaces loudly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hand-maintained versions had already drifted from the csproj (WPF-UI,
Magick.NET, PdfPig, and others). Remove the column and point readers to
Text-Grab.csproj / Tests.csproj as the single source of truth, and add the
Vortice.Direct3D11 / Vortice.DXGI packages introduced for HDR capture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduced a new user-scoped setting, `HiddenSmartPatternIds`, of type `System.String`. Updated `App.config`, `Settings.settings`, and `Settings.Designer.cs` to support storing and accessing this setting.
Added permissions in settings.local.json to allow web access to app.produckmap.com, read from local pdm skill directories, and execute pdm-related PowerShell and Bash commands for story maps, types, UI elements, and API operations. This enables broader interaction with ProDuckMap data and tools.
Add SaveHiddenSmartPatternIds_FileBackedModeWritesBothStores unit test to verify that saving hidden smart pattern IDs updates both in-memory settings and the backing file when file-backed mode is enabled, and that loading retrieves the correct values from both sources.
Renamed the "Regex Manager" button to "Patterns Manager" in the UI.
Enhanced PatternItem to support hiding built-in recognizer patterns, including an IsHidden property and PatternDisplay.
Updated the recognizer constructor and GetAll method to handle hidden patterns, allowing the Patterns Manager to show and unhide them as needed.
Added persistence for HiddenSmartPatternIds in SettingsService, including file registration, caching, load/save methods, cache invalidation, and a cloning helper for safe list management.
Renamed all UI and code references from "Regex Manager" to "Patterns Manager". Updated the manager window to display both user-saved regexes and built-in recognizers in a unified, read-only DataGrid. Added the ability to hide/unhide recognizers, with hidden items shown faded and italicized. Introduced a PatternItem abstraction for unified selection and management. Disabled direct add/delete in the grid; patterns are now managed via dialogs. Improved test and explain features to support both pattern types and handle invalid patterns gracefully. Updated comments and tooltips to match new terminology.
Add a custom style for Wpf.Ui DropDownButton to unify colors and hover/pressed states, importing the wpfui namespace. Also reformat bindings in SelectionStyleComboBoxStyle.xaml and the ResourceDictionary tag in SettingsStyles.xaml for readability. No functional changes outside the new style.
- Animate MoreOptionsToggleButton chevron in FindAndReplaceWindow and add tooltip
- Remove hardcoded Foreground and min row height in FindAndReplaceWindow
- Change RegExIcon to use Foreground instead of IconColor; remove IconColor property and binding
- Add PatternDropDownButtonStyle to SearchBar for better contrast in regex mode
- Update SearchBar to use new style and Foreground binding for RegExIcon
Improve visual feedback for NextStepDropDownButton by updating background and foreground colors based on selection and hover state, including icon color. Always show Freeze button in GrabFrame unless image is static.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants